Markovian Bias of Neural-based Architectures With Feedback Connections

Dynamic neural network architectures can deal naturally with sequential data through recursive processing enabled by feedback connections. We show how such architectures are predisposed for suffix-based Markovian input sequence representations in both supervised and unsupervised learning scenarios. In particular, in the context of such architectural predispositions, we study computational and learning capabilities of typical dynamic neural network architectures. We also show how efficient finite memory models can be readily extracted from untrained networks and argue that such models should be used as baselines when comparing dynamic network performances in a supervised learning task. Finally, potential applications of the Markovian architectural predisposition of dynamic neural networks in bioinformatics are presented.